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1 . WHAT IS IMPACT ASSESSMENT? 

In its broadest sense, impact assessment is the process of identifying the anticipated or 
actual impacts of a development intervention, on those social, economic and 
environmental factors which the intervention is designed to affect or may inadvertently 
affect. It may take place before approval of an intervention (ex ante), after completion (ex 
post), or at any stage in between. Ex ante assessment forecasts potential impacts as part 
of the planning, design and approval of an intervention. Ex post assessment identifies 
actual impacts during and after implementation, to enable corrective action to be taken if 
necessary, and to provide information for improving the design of future interventions. The 
stages in the project cycle where impact assessment needs consideration are shown in 
Figure 1. 

A distinction can be made between two separate but interlinked levels: 

• Internal monitoring and evaluation for ongoing learning, through for example the 
integration of specific impact indicators into existing management information systems, 
which makes information immediately available to staff; 

• External impact assessment, often involving independent investigators. Such 
assessments produce reports for specific purposes, such as poverty impact 
assessment, regulatory impact assessment, social impact assessment or health impact 
assessment. Certain types of ex ante assessment may be part of the approval process 
for certain types of intervention, including environmental impact assessment and 
economic impact assessment (cost-benefit analysis). These may contain their own ex 
post monitoring activities. Separate ex post assessments may be undertaken or 
commissioned for any particular intervention or set of interventions, to provide fuller 
information than may be available from routine monitoring and evaluation. 

In the context of sustainable development, the social, economic and environmental 
impacts of an intervention are all interlinked. The various types of impact assessment may 
therefore need to be combined in an integrated impact assessment, whose nature will vary 
according to the type of intervention, and the aims and cost-effectiveness of the overall 
impact assessment package. 

For each impact assessment type a wide range of methodologies has been developed, 
according to the precise purpose of the assessment, the types of question to be asked, the 
organisational context, the socio-economic context, available budget, research capacity 
and other factors. An impact assessment may include any or all of: 

• Quantitative statistical methods involving baseline studies, the precise identification 
of baseline conditions, definition of objectives, target setting, rigorous performance 
evaluation and outcome measurement. Such methods can be costly, limited in the 
types of impacts which can be accurately measured, and may pose difficulties for 
inference of cause and effect. Some degree of quantification may be necessary in 
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all impact assessments, in order to evaluate the success of the intervention and the 
magnitude of any adverse effects. 

• Qualitative methods suitable for investigating more complex and/or sensitive types 
of social impacts, e.g. intra-household processes, policy issues and investigation of 
reasons for statistical relationships and policy implications. These methods 
generally require high levels of skill, and may be relatively costly. Some degree of 
qualitative interpretation may be necessary in all impact assessments, in order to 
evaluate the causes of impacts which have been observed. 

• Participatory approaches suitable for initial definition or refinement of the actual or 
potential impacts which are of concern to stakeholders, questions to be asked, and 
appropriate frameworks and indicators to be used. Such approaches can contribute 
to all types of assessment, and are particularly suited to exploratory low budget 
assessments and initial investigation of possible reasons for observed statistical 
relationships. They offer a means of involving stakeholders in the research, 
learning and decision-making processes. These methodologies also require a 
certain level of skill, depending on the issues to be addressed and ways in which 
they are integrated with other methods. Some degree of stakeholder participation is 
likely to be necessary in all impact assessments, in order to achieve a good 
understanding of stakeholder perceptions of impacts. 

Whatever mix of techniques is used, consideration should be given to : 

• transparency and public accountability 

• stakeholder involvement 

• reliability of the information obtained 

• reliability of inference for policy improvement 

• cost and skill requirements 



For detailed discussion of quantitative, qualitative and participatory approaches, see 

Toolbox sections on: 

Programme Management Cycle 

Impact Assessment and Stakeholder Analysis 

Quantification Methods 

Qualitative Methods 

Participatory Methods 
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FIGURE 1. PROGRAMME CYCLE: 

AREAS WHERE IMPACT ASSESSMENT NEEDS CONSIDERATION 
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2. 



IMPACT ASSESSMENT FOR ENTERPRISE DEVELOPMENT 



Impact assessment for enterprise development (ED) has three different, but interrelated, 
objectives: 

• Accountability: to provide evidence about the achievements of ED interventions and 
their costs 

• Improving Programme/Project Effectiveness: providing recommendations about the 
means by which present and future programme/project performance could be improved 

• Policy Development: to provide guidance about the ways in which government and 
donor policies should be reformed so that the environment for enterprise development 
becomes more favourable. 

Each of these objectives is likely to shape the design of an IA in different directions. 
Accountability encourages a quantitative focus and the comparison of inputs with outputs 
and outcomes. Improving effectiveness encourages a focus on the processes by which 
inputs are converted into outputs and outcomes. Often this means that there is an 
extensive use of qualitative research methods. Policy development encourages a focus on 
macro-level contexts and often involves international comparisons. 

Impact assessment for ED at DFID is particularly important in showing how ED work 
contributes to poverty alleviation by benefiting the poor. Examples of this include support 
for small and medium enterprises (to create jobs or lower cost goods and services for the 
poor) and creating an enabling environment for private sector development (to promote 
pro-poor economic growth). 

The breadth of ED interventions means that very different approaches to IA have to be 
adopted, depending on the nature of the intervention. Methodologies are relatively well 
advanced at the micro-level (especially for microfinance) but are less mature at the macro- 
level of policy reform and institutional change. 

While all of the partners with whom EDAs work should, as a matter of good practice, be 
assessing the impacts of their activities, the Evaluation Department at DFID has a distinct 
role. It provides independent assessments of the achievements of DFID investments, 
including ED projects and programmes, and seeks to draw out policy lessons for DFID 
from comparative international studies. 
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3. PRINCIPLES OF IMPACT ASSESSMENT 



Impact assessments carried out as part of planning and approval of an intervention (ex 
ante) are predictive in nature, and generally follow well established methodologies (e.g. for 
environmental impact assessment, social impact assessment, health impact assessment, 
economic appraisal). 

Assessments carried out subsequently (ex post) aim to evaluate actual impacts. While 
techniques vary according to the nature of the intervention and the purpose of the 
assessment, the basic methodology is similar (Figure 2). 



Figure 2 General impact assessment methodology 
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• scope: determine which impacts should be investigated in the assessment; 

• targets: identify targets (where possible) for the impacts to be assessed, from the 
planning documents, or from widely accepted objectives appropriate for the type of 
intervention 

• indicators: identify indicators which will allow each impact to be measured, in relation to 
its target (if one has been identified); also methods of collection against these, and who 
will be responsible for collection 

• unplanned impacts: within the scope of the assessment, identify potentially significant 
unplanned impacts 
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• stakeholder identification: identify those social groups likely to be affected by planned 
or unplanned impacts, and other stakeholders (e.g. government bodies) with a 
significant interest 

• stakeholder involvement decide how stakeholders will be involved in the assessment; 
this may include involving stakeholders in refining the scope of the assessment, and in 
the identification of unplanned impacts and targets for planned ones 

• assessment of impacts: determine what impacts have occurred, their direct and indirect 
causes, and their importance in relation to targets 

• quantification of impacts: assess impact magnitude where practicable, in relation to 
targets 

• corrective action: define what steps can be taken to eliminate or reduce any significant 
adverse impacts or to compensate for them 

• policy learning: identify lessons for the planning and design of future interventions 

• reporting: document the findings of the assessment in a manner that is clearly 
understandable to those who will use them; identify uncertainties and reliability of 
findings; establish means of public access to the report 

• dissemination of findings: evaluation findings should be disseminated amongst 
stakeholders in a way that contributes to learning (e.g. by workshops, meetings, 
circulation of report); obtain stakeholder agreement to the report and agree follow-up 
action 

In many cases the assessment will be expected to assess impact on very broad goals, 
such as poverty alleviation. Unless the intervention can be expected to have a direct 
impact on such goals, it may be appropriate to identify relevant intermediary factors (e.g. 
education), and limit the assessment to impacts on them. The linkages between 
intermediary factors and broader goals can often be assessed reliably only through a 
complex policy-level impact assessment. In general, the targets and indicators used in the 
assessment will be those for which the intervention can be expected to have a direct 
impact. 

Whatever the precise scope of the assessment in relation to particular social, economic or 
environmental objectives, consideration should be given to the following potential issues: 

• time-dependency - might impacts that are small (or large) at the time of the 
assessment increase (or decrease) with time? 

• changing or abnormal conditions - how secure is an observed impact, in relation to 
economic or environmental shocks and other conditions which may vary from those 
pertaining at the time of the assessment? 

• cumulative effects - would a small effect become significant if the intervention or its 
effects were replicated? 

• remote effects - might unplanned impacts be occurring beyond the boundaries of the 
study area or community? 

• second order effects and interactions - might unplanned impacts be occurring that are 
not obviously associated with the intervention? 
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The last of these issues can entail a complex investigation of the interlinkages between 
social, economic and environmental impacts. A fully integrated impact assessment of this 
nature would be required if potentially important interactive effects are identified, within the 
scope of the assessment or subsequently (Figure 3). 



Figure 3 Types of Impact on Sustainable Development 
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4. CONCEPTUAL FRAMEWORK FOR ENTERPRISE AND PROJECT LEVEL 
IMPACT ASSESSMENT 



All impact assessment studies have an underlying conceptual framework. In well-planned 
and well-resourced lAs with long 'lead-in' times such frameworks are usually explicitly 
identified; by contrast, in many smaller scale exercises the framework is implicit and may 
be seen as 'common sense'. There are three main elements to a conceptual framework: 

• a model of the impact chain that the study is to examine 

• the specification of the unit(s) or levels at which impacts are assessed 

• the specification of the types of impact that are to be assessed. 



4.1 Impact Chains 

Behind all aid financed initiatives, is the assumption that the intervention will change 
behaviour and practice in ways that lead to the achievement (or raise the probability of 
achievement) of desired outcomes. lAs assess the difference in the values of key 
variables between the outcomes on 'agents' (individuals, enterprises, households, 
populations, policymakers etc) which have experienced an intervention against the values 
of those variables that would have occurred had there been no intervention (Figure 3). 
The fact that no agent can both experience an intervention and at the same time not 
experience an intervention generates many methodological problems. All changes are 
influenced by mediating processes (specific characteristics of the agent and of the 
economic, physical, social and political environment) that influence both behavioural 
changes and the outcomes in ways that are difficult to predict. 

The impact chain is very simply depicted in Figure 4. A more detailed conceptualisation 
would present a complex set of links as each 'effect' becomes a 'cause' in its own right 
generating further effects. For example, in a conventional microfinance project a package 
of technical assistance and capital changes the behaviour (and products) of a 
microfinance institution (MFI). The MFI subsequently provides different services to a 
client, most commonly in the form of a loan. These services lead to the client modifying 
her/his microenterprise activities which in turn leads to increased or decreased 
microenterprise income. The change in microenterprise income causes changes in 
household income which in turn leads to greater or lesser household economic security. 
The modified level of household economic security leads to changes in the morbidity and 
mortality of household members, in educational and skill levels and in future economic and 
social opportunities. Ultimately, perhaps, these changes lead to modifications in social 
and political relations and structures. 
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Figure 4 The Conventional Model of the Impact Chain 
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The complexity of such chains provides the assessor with a range of choices about which 
link (or links) to focus on. It is useful to distinguish between two main approaches with 
regard to which link(s) in the chain to focus on. For convenience, these are termed the 
'intended beneficiary' and the 'intermediary' approaches. 



The intended beneficiary approach, building on the ideas of conventional evaluation, seeks 
to get as far down the impact chain as is feasible (in terms of budgets and techniques) and 
to assess the impact on intended beneficiaries (individuals or households). The 
intermediary approach focuses purely on the beginning of the chain and in particular on 
changes in project outputs or institutional outreach and sustainability. The link between the 
intermediary process and the ultimate impact is often much wider than can be reasonably 
assessed for any individual project. One may therefore wish to do an 'intended beneficiary' 
assessment infrequently on a broad basis, to evaluate the policy assumptions underlying 
the design of individual interventions. For each intervention, one would only assess 
intermediary impacts. 



4.2 Units of assessment 

Following on from the design of a model of the impact path comes the choice of the unit(s) 
of assessment (or levels of assessment). Common units of assessment are the 
household, the enterprise or the institutional environment within which agents operate. 

The relative advantages and disadvantages of different units of assessment are 
summarised in Box 1. 
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Box 1 : Units of Assessment and Their Advantages and Disadvantages 

Unit Advantages Disadvantages 

Individual • Easily defined and • Most interventions have 

identified impacts beyond the 

individual 
• Difficulties of 

disaggregating group 
impacts and impacts on 

'relations' 

Definition and 
identification is difficult 
in microenterprises 
Much microfinance is 
used for other 
enterprises and/or 
consumption 
Links between 
enterprise performance 
and livelihoods need 
careful validation 



Sometimes exact 
membership difficult to 
gauge 

The assumption that 
what is good for a 
household in aggregate 
is good for all of its 
members individually is 
often invalid 



Community 


• Permits major 


• 


Quantitative data is 




externalities of 




difficult to gather 




interventions to be 


• 


Definition of its 




captured 




boundary is arbitrary 


Institutional Impacts 


• Availability of data 


• 


How valid are inferences 




• Availability of analytical 




about the outcomes 




tools (profitability, SDIs, 




produced by institutional 




transaction costs) 




activity? 


Household Economic 


• Comprehensive coverage 


• 


Complexity 


Portfolio (i.e. 


of impacts 


• 


High costs 


household, enterprise, 


• Appreciation of linkages 


• 


Demands sophisticated 


individual and 


between different units 




analytical skills 


community) 




• 


Time consuming 
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Enterprise • Availability of analytical • 

tools (profitability, return 
on investment etc) 



Household • Relatively easily defined • 

and identified 

• Permits an appreciation of 
livelihood impacts • 

• Permits an appreciation of 
interlinkages of different 
enterprises and 
consumption 



4.3 Types of impact 



An almost infinite array of variables can be identified to assess impacts on different units. 
To be of use these must be able to be defined with precision and must be measurable. 
Conventionally, economic indicators have dominated, with assessors particularly keen to 
measure changes in income despite the enormous problems this presents. Other popular 
variables have been levels and patterns of expenditure, consumption and assets. A strong 
case can be made that assets are a particularly useful indicator of impact because their 
level does not fluctuate as greatly as other economic indicators and is not simply based on 
an annual estimate. 

The social indicators that became popular in the early 1980s (e.g. educational status, 
access to health services, nutritional levels, anthropometric measures and contraceptive 
use) have recently been extended into the socio-political arena in an attempt to assess 
whether project interventions can promote empowerment. This has led to the 
measurement of individual control over resources, involvement in household and 
community decision-making, levels of participation in community activities and social 
networks and electoral participation. The bulk of this work has focused on changes in 
gender relations, but there are sometimes partially-formulated assessments of class 
relations within it. These extensions do add, however, to the complexity of IA work and 
require the skills of assessors who are experienced at making judgements on social 
relations. 

In addition, impact assessors should seek to keep the number of variables they measure 
to a manageable number and not be tempted to go for a comprehensive approach that will 
impact adversely on data quality and study relevance. 
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5. IMPACT ASSESSMENT IN PRACTICE 



Good practice impact assessments will be based on the principles identified in section 3. 
They must also seek to achieve a 'fit' with the objectives that are set, the intervention type 
and its goals, and the resources and time available (Figure 5). Inevitably, this entails 
compromises and trade-offs (e.g. if results are required rapidly then levels of rigour may 
need to be reduced). 



Figure 5 Achieving 'Fit' 
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5.1 Objectives of the IA: the objective(s) of the IA will shape the foci and methods of 
the study. An IA may focus on accountability, improving programme/project performance, 
policy development or a combination of these three objectives (Section 2). The greater the 
'mix' of objectives set, and the higher the levels of rigour to be achieved then the greater 
will be the need for resources and time. 
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5.2 Type and scale of ED intervention: ED intervention can take a variety of forms: 

• Fair trade 

• Microfinance 

• Regulatory frameworks 

• Business development services 

• Business associations and market linkages 

• Business and vocational skills development 

• Rural development projects 

• Appropriate and innovatory technology projects 

• Social and environmental enterprises 

• Informal sector development and support services 

• Tourism initiatives 

• Privatisation of state-owned enterprises and services 

The type of intervention has a significant influence on how an IA should be designed and 
who should conduct it. In addition, the greater the scale of intervention, then the greater 
will be the costs, or any given level of rigour and detail. 



5.3 Targets for the intervention: different ED interventions have different goals. Good 
practice demands that achievement is assessed against: 

• targets identified in the project framework 

• a broader set of targets - ideally the targets set out in DFID's Target Strategy Papers 
(Box 3). 
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Box 2 The DFID Target Strategy Papers are to be found at http://www.dfid.gov.uk/ 
then click on International Development Targets to obtain Adobe format of each TSP. 

• Halving poverty: this is central to EDDs work, incorporating its core enterprise 
development support to the legal and regulatory environment for private sector activity, 
business development services and microfinance then search International 

• Making government work for poor people: building stronger links between 
government and the private sector is essential to ensure both fulfil their responsibility to 
the public good 

• Meeting the urban challenge: this provides an opportunity to explore partnership 
arrangement with communities and enterprises which link physical improvements in 
infrastructure with the creation of employment and small enterprises 

• Poverty eradication and the empowerment of women: provides an opportunity for 
EDD to develop specific gender skills and understanding of how to further the gains in 
socio-economic empowerment, linked to EDD's knowledge of ED processes 

• Environmental sustainability and eliminating poverty: an opportunity to improve 
understanding of the environmental impact of enterprise development and its scope for 
enhancing business opportunities 

• Human rights for poor people: greater use of participatory techniques to enable people 
to better understand and exercise their rights 

• Addressing the water crisis: EDD has a potential role to support enterprises as users, 
suppliers or polluters of water 

• Better health for poor people: two components of this TSP are particularly relevant for 
EDD - health care financing, and the involvement of the private sector in health care 
delivery 

• Education for all: EDD has a role to play in developing a coherent approach towards 
educating for enterprise 



5.4 Resources for the IA: the rigour and quality of an IA are partly determined by the 
level of funds devoted to the IA and the quality of personnel. At present there are no clear 
guidelines on what levels of expenditure IA should attract, but good practice suggests that 
the level of investment in an IA should rise with the significance of the anticipated findings 
(e.g. (i) if a major expansion of a project is being considered then the initial project should 
be assessed in detail; (ii) if there is clear evidence that a project is failing and the main 
reasons are agreed upon by all key stakeholders then a low cost IA to verify this situation 
and confirm project termination may suffice). 

For many forms of intervention, and in many regions, the main resource constraint is 
skilled IA personnel. This often makes it necessary for EDAs to: 

• book IA consultants up well in advance of the IA 

• build IA timetables around IA consultant availability 

• seek to attach less experienced consultants to experienced IA consultants to broaden 
the human resource base. 

(see Box 3 for components that need to be taken into account in budgeting for IA) 

5.5 Timescale: careful consideration needs to be given to the timescale for an IA. 
Good practice suggests that the time available for an IA increases with: 
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• the scale of the ED intervention 

• the complexity of the impacts 

• the degree to which claims of impacts are challenged by different stakeholders. 

A common problem is that insufficient time is allowed for the IA to be conducted 
satisfactorily. IAs will be more effective if an initial data collection exercise precedes the IA 
itself. Ideally, all ED interventions should have a baseline survey (or baseline statement) 
against which impact assessors can compare their data. 
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Box 3: Budgets for Impact Assessment 

As many aspects of impact assessment should be part of the good practice 
management of a project, it is not always meaningful to try and separate out costs. 
However, in thinking through impact assessment in project design/implementation, 
the following components should be considered and, as far as practical, costed. 
These components and costs should be agreed with the partner-organisation, whose 
capacity to undertake the required level of IA (e.g. their existing MIS, staff 
understanding of IA) should be part of the appraisal process. 

INITIAL INFORMATION COLLECTION 

Baseline data: 

• What baseline data is required? 

• How will this will be collected? Who by? 

• Are there local skilled enumerators, or is local training needed, or are external 
consultants required? 

DESIGN OF IMPACT ASSESSMENT 

Stakeholder analysis: 

• What meetings are required to undertake stakeholder analysis? At these, 
consider: 

• Who should participate in IA and what methods of IA will be most effective in 
getting relevant information? What will be the costs to stakeholders of 
participation? 

• Are external facilitators required? Can internal staff be trained to do it? 

• What are the costs of these options? Which is likely to produce the better 
information? 

ONGOING INFORMATION COLLECTION 

• What steps are required to produce data that measures impact against 
indicators? 

• Is there an existing MIS that will generate relevant data? 

• Does MIS require modification/elaboration? 

• Do staff know how to use MIS, or is training required? 

• Is there a need for periodic large scale quantitative/qualitative exercises? Are 
there local personnel who can undertake these? 

ANALYSIS OF INFORMATION 

• Who is responsible for processing information and generating meaningful reports 
of progress against indicators? 

DISSEMINATION OF FINDINGS 

• What processes are required to disseminate finding? 

• Should there be series of workshops to involve stakeholders? 

• Should the report be translated into local language? 

• Who should the report be copied to? 

• Should the report be synthesised for publication in academic journals, at 
conferences etc? 



6. CONDUCTING AN IMPACT ASSESSMENT 



The level of impact assessment required, who it should be undertaken by, and the timing 
of any reports should be determined during the project design phase and set out in the 
Project Memorandum. There should, however, be some scope for flexibility to take 
account of any changes that may take place in the project's circumstances (e.g. major 
social, political or economic changes). The Logframe should highlight special features to 
be included in the impact assessment, for example vulnerable groups, the impact of a 
particular training programme, relative effectiveness of male and female staff in reaching 
female clients etc. Consideration of these issues at the design stage will enable much 
better assessment of the practical implications and learning of IA. 

It is important to engage the stakeholders, principally the implementing partners and their 
staff, from the beginning. Many project staff and managers feel threatened by impact 
assessment, believing their work is being judged in a critical manner and seeing it as a 
personal performance review. If they have been actively engaged in the design of the 
project from the start, however, and participated in the design of the IA strategy, fully 
understanding it's objectives, they will be more supportive. They should be encouraged to 
view it as a learning experience rather than judgmental. Whilst failure to meet targets due 
to negligence should not be overlooked or ignored, where impacts have not been as 
extensive as they might have been the priority should be to discover why rather than seek 
to blame, and then to consider what action can be taken to improve impact in the future. 

Depending on the nature and level of impact assessment required, it may be necessary at 
the start of the project, or even before, to contract out the collection of some baseline data 
on which to base future work. This may involve a range of qualitative and quantitative 
methods which can be time consuming and may require different skills from those 
possessed by project staff or EDAs. This should also identify which of these baseline 
indicators can be incorporated into existing Management Information Systems and how 
this should be done. It should also identify which types of information will require external 
consultants/researchers, or whether special staff will be needed to conduct research by M 
and E departments. 

For all their projects, EDAs are expected to oversee project reporting by project staff, and 
participate in routine monitoring and output to purpose reviews. Internal monitoring for 
ongoing programme level learning should take place through the integration of specific 
impact indicators into existing management information systems, however, which makes 
information immediately available to staff, and allows the project's management to act on 
that information more timeously. Periodic updates of this data would then be required 
(ideally undertaken by the same team) throughout the project. 

Regular reports (usually either quarterly or six monthly should be sent to EDAs by project 
staff. These reports should cover progress against activities and against outputs, using the 
impact assessment indicators specified in the project logical framework. This information 
should be gathered routinely by project staff. 

Periodic project monitoring visits are required by EDAs, the frequency of visits depending 
on the complexity and progress of the project. Monitoring visits should be used as 
opportunities to discuss the impact of the project with stakeholders and to verify the 
accuracy of project reports. 
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Output to purpose reviews are also required for all projects over £500 000. These may 
sometimes be done by EDAs and sometimes commissioned out, largely depending on the 
resources EDAs have available. These reports should focus on reporting against impact 
assessment indicators at the output level and at the purpose level. 

The key findings of the impact assessments at each level should be shared with the 
stakeholders involved to increase programme-level learning. Impact assessment should 
be regarded as a dynamic process and not as a series of static reports. Impact 
assessment can only contribute to lesson learning if the information is used as a basis for 
asking intelligent questions about project implementation and how it can be enhanced. 

The critical issues which the designer and manager of an impact assessment study will 
need to take into account are: 



• Costs and confidence 

The design of an IA must be very closely related to the budget available: this may be a 
platitude but overambitious designs continue to lead to poor quality studies or delays that 
make findings irrelevant. 

While rapid appraisal approaches may appear cheaper than large-scale surveys, however, 
rigorous qualitative lAs will require the use of high calibre staff who are given time to 
prepare properly, and the importance of engaging suitable staff should not be 
underestimated. 

Between these two extremes are a vast array of different options, and limited investments 
in project monitoring by program staff, for example by including in the project design and 
budget, development of an appropriate MIS system can make moderate cost impact 
assessment at high levels of quality much more feasible as less primary data collection is 
necessary 

• Availability of human resources for impact assessment 

In many, if not most, developing countries recruiting IA personnel who have the skills and 
qualities to interview, collate, analyse and write up findings is a key problem at both 
consultant and fieldworker levels. Commonly, different studies find themselves competing 
for the same small pool of people which, while it may usefully raise payments for scarce 
skills, puts these individuals under great strain and does not appear to stimulate a 'supply 
side response'. This must be recognised as a key constraint and efforts to build 'impact 
assessment' capacities professionally and institutionally should be a priority for 
development agencies if they intend to continue to emphasize the need for IA. 



• Respondents: motivation and representation 

The issue of how to persuade respondents to spare the time for an interview, and provide 
accurate and honest answers, is an important one that is rarely mentioned in IA 
methodological statements. Different strategies are needed for different types of 
respondent - program beneficiary, control group and program drop out. 
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Beneficiaries are the easiest group to approach as generally they accept 'answering 
questions' as one of the unavoidable transaction costs of being in a program, particularly 
one supported by a foreign donor agency. Motivation can be enhanced by having 
interviewers introduced by program officers: but, this has the danger of linking the 
assessor with field level staff and encouraging the recounting of 'the right answers'. For 
both data quality and ethical reasons the personal introductions that interviewers make 
prior to interview need to be carefully worked out so that respondents understand why they 
are being interviewed and have an opportunity to ask their own questions before the 
interview begins. 

Motivation is a more difficult issue with control groups as, having by definition no 
connection with a program, they have no incentive to cooperate. In many cases, however, 
the novelty value of being interviewed is sufficient encouragement (though expatriates 
should note that when they are working at a field site the willingness of people to be 
interviewed may be higher than is the norm because of the rarity value of foreigners). The 
problems of response increase significantly if longitudinal data is collected, as second and 
third interviews have much less perceived value. In such cases rewarding interviewees 
should be considered to promote data quality and for ethical reasons (what right have 
impact assessors to assume that the opportunity costs of an interview, particularly for poor 
people, are zero?). This can take the form of a social reward, such as bringing soda 
waters and snacks to share with respondents (this works well in East and Southern Africa), 
or 'bribery' where the interviewee is paid cash for surrendering her/his time. 

Program drop-outs represent a particular problem, and a failure to pursue drop-outs may 
have led to some lAs underestimating the negative impacts of projects. When the drop- 
out is traceable then significant effort is merited to obtain an interview/re-interview. Where 
drop-outs cannot be traced, or death has occurred, then a replacement respondent 
sampled at random from the original population, and preferably from the same stratum, 
should be interviewed. 

Participatory and rapid appraisal methods that work with groups generally manage to 
muster respondents because of the social interaction they create. However, care needs to 
be taken to observe who has turned up and, perhaps more significantly, who has not come 
to the meeting, and why. It is not necessarily the case that participants in a PLA exercise 
represent 'the community'. Additional interviews or focus groups may be necessary to 
collect information from people who do not turn up for communal PLA or RRA sessions. 



• The problem of 'low impact' impact assessments 

A final problem of IA concerns the impact of lAs on policy and practice. This depends in 
part on the original objectives of a study. It applies to both 'proving' and 'improving' lAs. 
The evaluation literature of the 1980s bemoans the limited influence of evaluation on 
subsequent decision-making. 

There are a number of ways this problem can be ameliorated: 

(i) Impact assessors need to devote more time to the 'use' of their studies (and 
perhaps a little less time to the product itself!). Their focus must go beyond 'the 
report' into a dissemination strategy aimed at decision-makers. Bullet point 
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summaries, short user-friendly papers, snappy presentations and strategic cups of 
coffee are the key to this environment. 

(ii) The timing of findings needs to be carefully considered. As a general rule of thumb 
the longer the length of time between data collection and presentation of findings, 
then the lower the impact for lAs focused on 'improving' practice. The common 
response to initial findings presented more than 9 months after completion of 
fieldwork is 'our program has already been redesigned so your findings have little 
relevance'. 

(iii) Program managers often regard impact assessors as impractical people who have 
lots of time on their hands. For high cost approaches pursuing the scientific method 
this will be of only limited significance as the people to whom one's results must be 
credible are in Washington and European capitals. However, for the vast majority 
of IA studies the issue of how to develop constructive relationships with program 
staff requires careful thought and action. Efforts to achieve co-ownership of findings 
by involving program staff in IA design, showing respect for their ideas and 
opinions, and discussing interim findings are ways of making influence more 
probable. 



• How 'robust' have the findings got to be? 

If an IA is to provide findings to a high degree of confidence (e.g. 95% confidence levels in 
statistical tests) then in most cases a 'complex approach' will be needed. This will be 
costly and time-consuming. By contrast, if an IA is required to provide corroboration of 
programme impact and strengthen aspects of implementation then a 'simple approach' 
should be adopted. 

The following approaches have been developed for micro-finance institutions, but are 
largely applicable to other enterprise projects, although measurable assessment of outputs 
and outcomes is harder to achieve in some other areas (e.g. impact of business 
development services), and will need to be well supported by qualitative and participatory 
studies. 



Simple Approach 

These are the most numerous forms of lAs. Reliability is moderate, at best (and based 
mainly on triangulation), and the major objective is to test the existing understanding of 
impacts and contribute to improvements in programme operation. The main audiences 
are programme managers and donor 'country-based' staff. The central methodological 
feature of such an approach is the use of a variety of methods. Usually this involves a 
small scale client survey, compared with a comparison group that could be rapidly 
identified (e.g. approved clients who have not yet received services) and cross-checked by 
rapid or participatory appraisal methods. If a baseline study is not available then a recall 
methodology would be utilised. The key variables to be studied would depend on 
programme objectives, but for easily quantifiable variables (e.g. income and assets) the 
focus would be on ordinal and nominal measurements. For programmes prioritising 
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empowerment goals and local institutional development, then participatory methods would 
be highlighted and the survey work might be dropped altogether. 

A Moderate Approach 

The moderate approach would involve substantially more costs than the simple approach, 
would yield higher levels of reliability (statistical inference rather than triangulation) but is 
not likely to deliver findings for a period of 2 to 3 years. Its focus is on both proving impact 
and improving programmes. Its audiences would include policymakers (looking for 
reassurance about their agency's investments) and the senior managers of programmes. 
The 'mix' would centre on a significant survey that would stratify clients and compare them 
with a carefully matched control group. The survey would involve at least two visits with a 
minimum of 12 months between them and recall techniques would not be used. 
Contextual and cross-checking materials would be produced by rapid appraisal 
techniques, and carefully planned participant observation and case studies might also be 
commissioned. The selection of variables would depend on programme objectives but is 
likely to be more extensive than for the simple approach, and measurement would focus 
on interval and nominal scales. 

A Complex Approach 

The complex approach focuses on ensuring high levels of reliability with regard to the 
attribution of causality and has an exclusively 'proving' orientation. Its main audiences are 
policymakers and researchers and it is likely to be 4 to 6 years after launch before findings 
are available. The central method in such an approach is a large scale sample survey 
very carefully constructed to represent all key features of the client population. This is 
compared against a carefully selected control group, so that the number of households 
surveyed is likely to be between 750 and 1500. At least 3 interviews will be conducted 
with each household over a period of 2 to 3 years. A wide set of variables will be 
measured and the focus will be on high precision through interval measurements. A set of 
related studies on institutional performance would be conducted, but the heart of the study 
would be the econometric analysis of survey findings. 



• Summary 

The key task for the IA designer is to select an approach that can meet the objectives of 
the specific assessment at an acceptable level of rigor, that is compatible with the 
program's context, feasible in terms of costs, timing and human resource availability and 
that avoids the problems identified in earlier sections. Wherever possible an IA 
methodology should be piloted before full implementation. 

The questions that s/he must answer can be summarised as follows: 

• What are the objectives of the assessment? 

• How is the information to be used and by whom? 

• What level of reliability is required? 

• How complex is the program, what type of program is it, what is already known about it? 

• What resources (money, human and time) are available? 
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7. COMMISSIONING AN IMPACT ASSESSMENT: TERMS OF REFERENCE FOR 
CONSULTANTS 



EDAs may wish to commission external impact assessments where: 

- A complex IA has been decided at the design stage involving a range of quantitative, 
qualitative and participative methods 

- Where a project is not achieving the stated outputs and a greater level of analysis is 
needed to determine the reasons 

- Where an EDA wishes to look at IA across a sector or theme (e.g. outreach to the 
urban poor) to draw comparative conclusions 

Terms of Reference should be drawn up in participation with project staff and with 
professional support from DFID colleagues in other sectors. Costs of the IA strategy 
should have been included in the project's budget at the design stage; these should have 
included capacity for the project to absorb the lessons that may emerge from the study, for 
example to hold workshops with staff to disseminate the findings and negotiate changes in 
the organisation's work that will lead to improved impact in the future. 

Detailed guidance on the process for commissioning an impact assessment is contained in 
DFID's Evaluation Guidelines. 1 The following is a summary of the main points to be 
included in the Terms of Reference, which can then be used as the EDA's guide for 
steering the IA through to satisfactory completion. 



Model Terms of Reference for Consultants 

1 Project Title: Brief Description 

2 Background 

This should provide general background and context for the proposed consultancy: 

• reasons for the consultancy (including details of problems/constraints faced by the 
recipient) 

• the level of local resources/capacity available to support the consultancy 

• the proposed role of the consultants 

• arrangements for working with local staff (including role of counterparts and/or local 
task forces) and some idea of the existing MIS and what data is already available. 

3 Overall Objectives 

Clearly state the objectives and intended users of the impact assessment. Ideally the 
objectives should include both justifying programme investments and improving 
programmes. 

4 Scope and Methods of the Work 

This section should list in detail all of the tasks and activities to be carried out by the 
Consultant in the sequence that they are expected to be undertaken. 



1 Available from the Monitoring & Evaluation Dept (link to DFID M&E site). 
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State the key research questions. These should be guided by the intended use of 
the information that is to be gathered, and limited to those which it is most 
important to address 

Identify the key partners in the IA process and the stakeholders to be included in 
the sample - this will affect methodology. 

Determine the method or mix of methods envisaged for the IA. The choice of 
methods should be based on the objectives of the impact assessment, the key 
research questions, the availability of resources (time, money and people) and the 
degree of generalisation and precision required. Involving project staff in the design 
process is very important for improving the credibility and ultimate usefulness of an 
IA. Their knowledge of the programme and the context can be invaluable in 
informing many aspects of the research design. 

Checklist: 

• Are the methods proposed consistent with the time and resources 
available for the impact assessment? 

• Will the methods provide the type and quality of impact assessment 

required by the stakeholders ? 

• Have specific questions or hypotheses relating to each impact 
assessment indicator been generated during the inception stage of the 
impact assessment ? 

• Will the methods to be used by the impact assessors provide valid and 
reliable information which will allow these questions to be answered ? 

• Are the methods to be used clearly described in the proposal and 
inception report ? 

5 Expected Outcome and Deliverables 

The general outcome should be described. Against each of the tasks and activities set out 
under the scope of work there should be a corresponding output. It is important that 
consultation of the draft report with stakeholders is included, to assist in developing 
ownership of the findings. Not all stakeholders will necessarily agree with the findings; this 
should be reflected in the report. Dissemination of the report should also be included as a 
key outcome. 

There will be occasions when the partner organisation feels the outcome of the IA is not a 
fair reflection of their achievements or did not take into account some special 
circumstances. This can be a source of tension, even conflict. It is most likely to happen 
when the project staff and management feel their views were not taken into account in 
determining indicators and criteria for assessment, and confirms the need to engage them 
in design of the IA strategy from the start. 2 



2 DFID's Centre for Social Dimensions of Business Practice is developing guidelines for dealing with dissent 
(link required) ??Link URL?? 
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6 Competency and Expertise Requirements 

Minimum requirements and preferences should be stated, and CVs requested for the 
consultants put forward to carry out substantial parts of the work. You should expect them 
to advise of their: 

• professional expertise relating to the issue being evaluated 

• knowledge of the country / region 

• general development expertise and experience 

• cross-disciplinary skills (e.g.. social, economic and institutional) 

• awareness of gender and cultural issues 

• impact assessment experience, and of what methodologies 

• any special language skills 

7 Conduct of the Work 

This should set out any useful information about the way the consultancy is expected to be 
organised and implemented. In particular it should state what will be: 

• the role of the consultancy team leader; 

• the role of team members and the organisation of the team; 

• the design and implementation of work programmes; 

• target dates for completion of work programmes; 

• the role of local counterpart staff in the conduct of the work 

• how the research process can benefit those involved 

• how the research process will contribute to the programme 

8 Reporting Requirements 

This covers both routine reporting on the progress of the assignment and on the final 
outcome/ conclusions of the work carried out. 
Particular attention should be paid to: 

• the scope and timing of progress reports for DFID and recipients 

• the need for presentations/ workshops to discuss progress and conclusions with project 
staff and management 

• the coverage and timing of reports, setting out the results of the consultancy. 

Impact assessment reports need to be short, clear, and easy to read if they are to be 
accessible and effective. Authors should write for a general audience, and should bear in 
mind that English will not be the first language of many readers. Technical and academic 
jargon should be avoided. Long reports will not be read, and are expensive to translate. 
At a minimum the report should include: 

A clear statement of the objective of the impact assessment 
A description of the programme including its objectives 

A description of the methodology used for qualitative and quantitative components, 
especially criteria and process for selecting participants 
Findings, interpretation of results, conclusions and when appropriate, 
recommendations for programme improvement 
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Dissemination of Findings 

Impact assessment findings will need to be effectively disseminated within DFID and its 
partners if the aim is policy lesson learning. Accountability requires that the impact 
assessment findings are accessible to the public and participants in the UK and partner 
countries. 

For this to be the case : 

• dissemination needs to be planned from the beginning, and included in the TOR. 

• dissemination needs to be directed at, and tailored for, specific groups. 
Dissemination should be project / audience-led, not consultant- or product-led. 

• reports (or at least summaries) should be translated into local languages where 
necessary (and the costs of this included in the project budget). 

• findings need to be openly and widely disseminated throughout the process, not 
just at the end. 

• the budget needs to contain sufficient resources for effective dissemination. 

Dissemination methods need to be designed and implemented on a case by case basis. 
Three primary methods of dissemination can be used : 

• presentations, workshops and seminars 

• impact assessment reports and summaries 

• articles in general and academic publications 
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8. COMMON IMPACT ASSESSMENT METHODS 



Table 1 (below) sets out the methods most often used in lAs, and Table 2 (on the next 
page) summarises the comparative strengths and weaknesses of these methods. 



Table 1 : Common Impact Assessment Methods 
Method Key Features 



Sample Surveys 



Collect quantifiable data through 
questionnaires. Usually a random 
sample and a matched control group 
are used to measure predetermined 
indicators before and after intervention 



Rapid Appraisal 



A range of tools and techniques 
developed originally as rapid rural 
appraisal (RRA). It involves the use of 
focus groups, semi-structured interview 
with key informants, case studies, 
participant observation and secondary 
sources 



Participant Observation 



Extended residence in a program 
community by field researchers using 
qualitative techniques and mini-scale 
sample surveys 



Case Studies 



Detailed studies of a specific unit (a 
group, locality, organisation) involving 
open-ended questioning and the 
preparation of 'histories' 



Participatory Learning and Action 



The preparation by the intended 
beneficiaries of a program of timelines, 
impact flow charts, village and resource 
maps, well-being and wealth ranking, 
seasonal diagrams, problem ranking 
and institutional assessments through 
group processes assisted by a facilitator 
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Table 2: 



Comparative strengths and weaknesses of different methods 



Method Surveys Rapid Participant Case Participatory 

Criteria appraisal observation studies Learning and 

Action 



1 . Coverage 

(scale of High Medium Low Low Medium 

applicability 



2. Represent- High Medium Low Low Medium 

ativeness 



3. Ease of data 

standardisation, High Medium Medium or Low Low Medium or low 

aggregation and 
synthesis (e.g. 
quantification) 



4. Ability to 

isolate and High Low Low Low Low 

measure non- 
project causes of 
change 



5. Ability to cope High Medium Medium Medium Medium 

with the 

attribution 

problem 



6. Ability to Low High High High High 

capture 

qualitative 

information 



7. Ability to Low High High Medium High 

capture 

causal processes 



8. Ability to Minimal Medium High Medium Medium 

understand 

complex 

processes (e.g.. 

institution 

building) 



9. Ability to Low High Very high Medium High 

capture 

diversity of 

perceptions 
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Method Surveys Rapid Participant Case Participatory 

Criteria appraisal observation studies Learning and 

Action 



10. Ability to elicit 
views of women 
and 

disadvantaged 
groups 


Low 


Medium 


High 


High 

(if 

targeted) 


Medium 


11. Ability to 
capture 
unexpected or 
negative 
impacts 


Low 


High 


Very high 


High 


High 


12. Ability to 
identify 

and articulate felt 
needs 


Low 


High 


High 


Medium 
(due to 
low 
coverage) 


High 


13. Degree of 
participation 
encouraged by 
method 


Low 


High 


Medium 


Medium 


Very high 


14. Potential to 
contribute to 
stakeholder 
capacity 
building 


Low 


High 


Low 


Medium 
to low 


Very high 


15. Probability of 
enhancing 
downwards 
accountability 


Low 


High 


Medium 


Medium 


High 


16. Human 

resource 

requirements 


Specialist 
supervision 
- large 
numbers of 
less qual- 
ified field 
workers 


High skilled 
practition- 
ers,who are 
able to rite 
-up and 
analyse 
results 


Medium skilled 
practitioners, with 
good 

supervision, who 
are prepared to 
commit for 
lengthy period 


Medium 
skilled 
practit- 
ioners with 
good 
super- 
vision 


High skilled 
practitioners 


17. Cost range 


Very high 
to medium 


High to 
medium 


Medium to low 


Medium 
to low 


High to Medium 


18. Timescale 


Very high 
to medium 


Medium to 
low 


High 


High to 
medium 


Medium to low 




▲ 

Surveys 


▲ 

Rapid 
appraisal 


▲ 

Participant 
observation 


▲ 

Case 
studies 


▲ 

Participatory 
learning and action 



Source: Adapted from Montgomery et. al. (1996). 
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As mentioned in Sectionl, IA methods fall into three broad categories: qualitative, 
quantitative, and participatory. Techniques from each of these categories can be combined 
in any one IA, always bearing in mind the purpose of the IA. 

Quantitative approach 

This most closely resembles a 'scientific method', but even with careful design is still 
subject to problems such as: 

• sample selection bias 

• mis-specification of underlying causal relationships, and 

• respondent motivation 

Selection bias may occur because of: 

i. difficulties in finding a location at which the control group's economic, physical and 
social environment matches that of the treatment group 

ii. the treatment group systematically possessing an 'invisible' attribute which the control 
group lacks (most commonly identified as entrepreneurial drive and ability) 

iii. receiving any form of intervention may result in a short-term positive response from the 
treatment group 

iv. the control group may become contaminated by contact with the treatment group. 

Problems (i) and (iv) can be tackled by more careful selection of the control group. This 
applies particularly to controlling for access to infrastructure (which has a key influence on 
input and output prices as well as other variables) and ensuring that the control group is 
located far away from the treatment group. Problems (ii) and (iii) are more intractable, but 
in many cases they can be tackled by using program-accepted 'clients-to-be', who have 
not yet received the services, as the control group. It must be noted, however, that this 
approach will not be valid when the take up of services is based on diffusion through a 
heterogeneous population. 

The mis-specification of underlying causal relationships arises most commonly 
because of the assumption that causality is a one-way process. Overcoming this can be 
enormously demanding in terms of data requirements, technical expertise and costs, and 
is often only feasible on very rare occasions. The main means of dealing with it are (i) 
tracing dropouts from both the treated and control groups; (ii) only conducting lAs on 
relatively mature programs; (iii) interim impact monitoring activities to gather qualitative 
information about the complexity of causality; and (iv) retrospective, in-depth interviews 
with clients. 

Respondent motivation operates at two levels. Firstly, the people who take up the 
services may be more strongly motivated to participate in the intervention than those in the 
control group (similar to 'the entrepreneurial factor' referred to above). Secondly, their 
motivation when taking part in the IA, whether it be completion of a survey form or in the 
conduct of an interview, may be influenced by their desire to say what they think the 
interviewer wants to hear, or how they relate to the interviewer, or the time they are able to 
give to completion of the form, and so on. Training of interviewers to ask questions in a 
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similar manner, and careful design of forms may help to some extent but it is impossible to 
completely overcome this problem 



Qualitative Approach 

The qualitative approach attempts to resolve some of the problems described above by 
seeking to provide an interpretation of the processes involved in intervention and of the 
impacts that have a high level of plausibility. It recognises that there are usually different, 
and often conflicting, accounts of what has happened and what has been achieved by a 
program. The validity of specific lAs adopting this approach has to be judged by the 
reader on the basis of: 

(i) the logical consistency of the arguments and materials presented; 

(ii) the strength and quality of the evidence provided; 

(iii) the degree of triangulation used to cross-check evidence; 

(iv) the quality of the methodology; and 

(v) the reputation of the researcher(s). 

Although such work has been common in development studies for decades, it is only 
during the 1980s that its relevance for IA has been recognised. This recognition has arisen 
partly because of the potential contribution of qualitative approaches (especially in 
understanding changes in social relations, the nature of program staff-beneficiary relations 
and fungibility) and partly because of the widespread recognition that much IA survey work 
was based on inaccurate information collected by questionnaire from biased samples. 
Low budget and low rigour lAs claiming to adopt the scientific method were at best 
pseudo-science, but more often simply bad science, despite the sophisticated analytical 
tools that were applied to poor datasets. 

However, lAs with their roots in the humanities have considerable difficulties with regard to 
the attribution of cause and effect. Such studies cannot usually demonstrate the causal 
link as they are not able to generate a 'without program' control group (although at times 
some researchers neglect to mention this to the reader and simply assume causality). 
Instead, causality is inferred from the information about the causal chain collected from 
intended beneficiaries and key informants, and by comparisons with data from secondary 
sources about changes in out-of-program areas. Problems also arise because not 
infrequently the labels 'rapid appraisal', 'mini-survey' and 'case study' are applied to work 
which has been done in an ad hoc manner and does not achieve a minimum professional 
standard in terms of informant selection and the rigor of data collection and analysis. 
Examples of this include: i) basing data collection only in program areas that are 
performing well, and surveying best clients, and ii) inferring that the data collected in one 
area applies to all clients without explaining this assumption. 

While such studies cannot provide the degree of confidence in their conclusions that a fully 
resourced scientific method approach can yield, in some cases their conclusions may be 
more valid than survey based IA work that masquerades as science but has not collected 
data with scientific rigor. Whatever, it is becoming increasingly common to combine 
'scientific' and 'humanities' approaches so as to check the validity of information and 
provide added confidence in the. In the future dealing with attribution by multi-method 
approaches seems the way forward. 
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Participatory Approaches: PLA (Participatory Learning and Action) 

In the last five years participatory approaches to development planning and management 
have moved from being a fringe activity to centre stage. While many donor agencies, 
have simply added a bit of PLA to their existing procedures, it can be argued that this is 
inappropriate as conceptually participatory approaches challenge the validity and utility of 
the scientific method as applied to developmental problems. 

According to this line of argument the scientific/quantitative method fails as: 

• it ignores the complexity, diversity and contingency of winning a livelihood; 

• it reduces causality to simple unidirectional chains, rather than complex webs; 

• it measures the irrelevant or pretends to measure the immeasurable; and, 

• it empowers professionals, policymakers and elites, thus reinforcing the status quo and 
directly retarding the achievement of development goals. 

Supporters of participatory approaches do not believe that ultimately there is one objective 
reality that must be understood. Rather, there are multiple realities and before any 
analysis or action is taken the individuals concerned must ask themselves, 'whose reality 
counts?'. Their answer is that the perceived reality of the poor must take pride of place as, 
if development is about 'empowering the poor' or 'empowering women' (as virtually all 
development agencies now say), then the first step towards empowerment is ensuring that 
'the poor' or 'women' take the lead in problem identification and analysis and knowledge 
creation. 

For impact assessment the purist PLA line is damning: conventional baseline surveys are 
virtually useless for impact assessments, and the question now is about how widely local 
people can be enabled to identify their own indicators, establish their own participatory 
baselines, monitor change, and evaluate causality. By this means two objectives may be 
achieved: 
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(i) better impact assessments, and 

(ii) intended beneficiaries will be empowered through the research process itself 

In practice, the art of participatory impact assessment (PIA) is in its infancy and a 
pragmatic rather than a purist approach has been common. 

The reliability of participatory methods varies enormously, as with 'scientific' surveys, 
depending largely on the motivation and skills of facilitators and those investigated 
and the ways in which informants' perceptions of the consequences of research are 
addressed. Nevertheless, it is argued that a number of rigorous comparative studies 
have shown that, when well-conducted, participatory methods can be more reliable 
than conventional surveys. 

To date the literature on PLA and PIA has only partially addressed the issue of 
attribution. From a scientific perspective PIA has grave problems because: 

• of the subjectivity of its conceptualisations of impact; 

• the subjectivity of the data used to assess impact; 

• the variables and measures used vary from case to case and do not permit 
comparison; 

• its pluralist approach may lead to a number of mutually conflicting accounts 
being generated about causality; and, 

• the assumption that because lots of people are taking part in an exercise 
means that all are able to 'voice' their concerns (so that opinions are 
representative) is naive about the nature of local power relations. 

From the perspective of a 'new professional', however, such a set of accounts is 
unproblematic, as it reflects the complexity and contingency of causality in the real 
world. In addition, it can be argued that PIA contributes to program goals (perhaps 
particularly in terms of empowering women and the poor) by not facilitating the 
continued dominance of target groups by powerful outsiders. 

Because of the different pattern of strengths and weaknesses offered by each 
method of IA, there has been a growing consensus amongst impact assessors that 
the central question is no longer "what is the optimal method for this study" but rather 
"what mix of methods is most appropriate for this study, and how should they be 
combined." 

Depending on the level of resources available and the context, impact studies 
increasingly seek to combine the strengths of different approaches and, in particular, 
seek to combine the advantages of sample survey and statistical approaches 
(representativeness, quantification and attribution) with the advantages of humanities 
or participatory approaches (ability to uncover processes, capture the diversity of 
perceptions, views of minorities, unexpected impacts etc). 
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In well resourced studies with long time scales all of these different methods may be 
utilised in a comprehensive fashion. In cases where a high degree of statistical 
confidence is required (for example, when it is desired to 'prove' impact for policy or 
major investment purposes) then a large scale, longitudinal sample survey must be 
mounted, preferably supported and triangulated by the use of their methods on a 
limited scale. By contrast, if an IA is required to provide independent corroboration 
of the impact of a small scale program and strengthen aspects of its implementation 
then a mix of rapid appraisal and small scale survey is likely to be appropriate. 

Table 3 summarises the conditions in which key methods are, and are not, 
appropriate. 
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Table 3: 



Conditions in which key methods are, and are not, appropriate 



Source: Adapted from Montgomery et al 1 996. 



Sample Surveys are appropriate when: 



A project affects large numbers of beneficiaries 

Policymakers require accurate estimates of project impacts 

Statistical comparisons must be made between groups over 
time and/or between locations 

Project delivery/implementation mechanisms are operating 
well, thereby justifying investment in the assessment of 
impacts 

The target population is heterogeneous and it is difficult to 
isolate the influence of factors unrelated to the project (e.g. 
contextual variables, other programmes etc) 



Rapid Appraisal and/or PLA are appropriate when: 



The project is adopting or promoting participatory 
principles in (re-)planning, implementation, monitoring 
and evaluation 

An understanding of the motivations and perceptions of 
project clientele is a priority 

One of the purposes of the study is to assess whether or 
not felt needs are being addressed by the project 

The impact of community-based organisations or other 
institution building activities are of importance 

There is a need to understand the quality of other data 
collected through surveys 



Participant Observation and/or Case Studies are 
appropriate when: 

An understanding of the motivations and perceptions of 
project clientele is a priority 

Other methods (surveys and rapid appraisals) are unlikely to 
capture the views of minorities or women 

One of the purposes of the study is to assess whether or not 
felt needs are being addressed by the project 

The impact of community-based organisations or other 
institution building activities are of importance 

There is a need to understand the quality of other data 
collected through surveys or rapid appraisals (e.g.. causal 
processes) 



Sample Surveys are usually not appropriate when: 



A project affects small numbers of beneficiaries 

Policymakers are mainly concerned with project outcomes. 
(Was infrastructure completed on time and within budget? 
How many people us the health clinics?) 

Project implementation is recent and untested, and it is likely 
that the way in which the project is implemented will have 
little impact at the present time. 

The purpose of the assessment is to study and evaluate 
complex activities or processes (e.g.. the development & 
operation of community-based organisations; qualitative use 
of skills as a result of training programmes) 

The purpose of the assessment is to document easily 
observable changes in the physical environment or other 
tangibles (which can be assessed through simpler, structured 
visits) 

The purpose of the assessment is to understand whether or 
not the project is meeting the felt needs of project clientele 



There is a need for contextual studies before designing 
more complex monitoring or impact assessment 
exercises (e.g.. case studies, or surveys). 



Rapid Appraisal and/or PLA are not appropriate 
when: 

Projects are relatively un-complex, in which bounded 
locations are not units of analysis (e.g.. Health centres 
serving a wide catchment area, or large schools serving a 
variety of communities) 

Indicators of project impact are uncontroversial, and 
negative impacts are unlikely 

Standardised and statistically representative 
generalisations for a large and diverse project population 
are regarded as the sole priority 

Participation of clientele is not a priority (e.g.. in public 
administration or power sector reform, or an 
organisational change programme - in these types of 
projects more limited focus group discussions with staff 
may be more appropriate) 



There is a need for contextual studies before designing more 
complex monitoring or impact assessment exercises (e.g.. 
before carrying out rapid appraisals, or before designing a 

survey) 

Participant Observation and/or Case Studies are usually 

not appropriate when: 

The project is small and "uncomplicated", providing a specific 
service or limited intervention which is unlikely to affect 
community dynamics beyond a few specific effects (e.g.. 
disease specific health facilities or campaigns) 

Bounded locations are not units of analysis (e.g.. health 
centres serving a wide catchment area, or large schools 
serving a variety of communities) 

Indicators of project impact are clear and easily measurable 
or assessable (by survey or rapid appraisals) 

Indicators of project impact are uncontroversial, and negative 
impacts are unlikely 

Information is needed quickly, and standardised, 
representative 

(statistical) generalisations are regarded as the sole priority 
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